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Competencies in graduate education: What should students know about 

research and statistics? 

Abstract 

University faculty identified six broad topic areas in research and statistics that 
they considered important for master's level graduate students. These broad 
areas were further defined by 31 more specific topics. Test items were 
developed and administered to graduate students in introductory research 
methods courses. We found that, after taking an introductory research 
methods course, master's students were generally prepared to critique 
research but not to conduct independent, original research. 
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Competencies in graduate education: What should students know about 

research and statistics? 

What should master's level graduate students in colleges of education 
know in the area of research and statistics? Many graduate students pursuing 
a master's degree are required, or make the choice, to write a thesis. Others 
are required to produce a document that demonstrates their skills in 
synthesizing research on a suitable topic that applies to their particular field. 
Still others are asked to convince a faculty committee that they understand 
research and statistics as they apply to their field by performing on written 
and oral comprehensive examinations. Are students adequately prepared in 
their programs of study to meet these outcomes? The purpose of this study 
was to identify those research skills that are considered important by the 
teaching faculty and to assess the degree to which students in master's 
programs in education are acquiring those skills as a result of coursework. 

This is the initial phase in a larger study in which our ultimate goal is to 
provide some standards in the area of research and statistics for colleges of 
education. Our vision is that faculty in colleges of education can reach 
consensus regarding the skills in research and statistics desired as outcomes in 
graduate students. 

Literature Review 

Skills needed for research 

Graduate students pursuing master's degrees are expected to develop 
skills that will enable them to critically read and evaluate research and to 




4 



Research Competencies in Graduate Education 

4 

identify suitable research problems. Students who pursue a thesis option 
must be able to design and conduct a study and to analyze and interpret their 
findings. In addition, whether or not they plan to write a thesis, they must be 
able to communicate their findings to an audience who will, in turn, critically 
evaluate their work. The complete research processes involved in scholarly 
research activities include both technical skills, such as organizing literature 
searches and writing in a scholarly fashion, and critical thinking skills, such 
as proposing thoughtful and appropriate hypotheses based on and extending 
an existing knowledge base. According to Ary, Jacobs, and Razavieh (1996), 
desired competencies for beginning researchers include developing viable 
research problems; understanding the role of previous research and theory; 
using sampling procedures, statistics, and measurement that are appropriate 
for selected research problems; recognizing the advantages and disadvantages 
of differing approaches to research; interpreting research findings; and 
preparing a final report in a scholarly format. Although all graduate students 
may not be required to conduct original research, at the very least they will 
certainly be expected to evaluate the research of others'. 

Student preparation 

Master s level graduate students often take a single introductory research 
methods course to meet the requirements for their degree (Doak, 1982). 

Given that the course may be the student's only training in research and 
statistics, teaching faculty are relied upon to deliver a comprehensive and 
important curriculum. They must communicate the broad spectrum of 
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research in an educational setting, as well as impart sufficient specific 
knowledge that students will need to conduct research independently. 

Surveys have assessed the effectiveness of this course and found the course 
generally useful in providing students with basic skills such as evaluating 
research and understanding terminology (Todd & Reece, 1987). However, 
other studies have shown that both graduate students and faculty question 
the adequacy of the preparation students receive from such a course, 
especially if the course is to serve as motivation for conducting independent 
research (Monahan, 1994). Sipe and Doherty (1993) found, in a case study 
where graduate students conducted real research in a community setting, that 
the traditional training graduate students receive is not adequate in preparing 
them to engage in meaningful research. Indeed, it appears to benefit students 
a great deal if they are involved in other research activities, such as 
mentoring by faculty, conference presentations, collaboration with other 
students in research teams, and the like (Ertmer, Provo. Moreno, Butcher, 
Leuck, MacDougall, Newby, 1996; Green & Kvidahl, 1990). However, our 
study examined only the benefits students gained from a single introductory 
research and statistics course. 

Need for standards 

Is it important to establish competencies in research and statistics for 
graduate students? Professionals in education-related associations agree on 
the importance of competencies, or standards, and encourage their 
development. For example, The American Federation of Teachers, the 
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National Council on Measurement in Education, and the National Education 
Association developed a document titled "Standards for Teacher Competence 
in Educational Assessment of Students" (1990). These associations were 
committed to the ideal that assessment is an integral part of good teaching 
and that creating standards would serve as a guide in teacher preparation 
programs. In their document they define a standard as "a principle generally 
accepted by the professional associations responsible for the document." The 
National Council of Teachers of Mathematics developed (1989), and is in the 
process of revising, their standards for the teaching of mathematics in much 
the same way . In the field of educational research, the American Educational 
Research Association (1991) proposed a set of ethical standards to serve as a 
guide in the consideration and treatment of human subjects. However, 
standards for the content and mastery of outcomes in research and statistics in 
graduate programs have not yet been proposed. 

Method 

Data were collected from faculty and graduate students associated with 
colleges of education in two universities located in the western United States. 
Faculty were those involved in teaching research and statistics methods 
courses at the graduate level. Graduate students were enrolled in master's 
programs in education and were taking an introductory research methods 
course. The research methods courses were considered by the teaching faculty 
to be very similar in content at the two universities. 



Research Competencies in Graduate Education 

7 



The teaching faculty (n=5) collaborated on identifying six broad topical 
areas in research and statistics that they determined should be addressed in 
developing outcomes of a master's program in a college of education. The six 
areas were: types of research, research process, hypothesis testing, data 
gathering, descriptive statistics, and inferential statistics. The six broad areas 
were further defined by 31 underlying topics. Based on the 31 topic areas, they 
designed three test items for each topic to assess student mastery. For 
example, the following three test items were designed to assess an 
understanding of sampling: 

(Item #1) A researcher plans to conduct a poll by sampling fans from the 
population of all fans of the Rockies. She selects her sample by stopping 
people as they leave the stadium upon completion of a game. The type of 
sampling she used is 

a. systematic 

b. stratified 

c. convenience 

d. random 

(Item #2) A sample is considered random if 

a. every subgroup in the population is included 

b. every element in the population has an equal and independent 
chance of being included 

c. an infinite number of subjects is available for sampling 

d. the sample is known to well represent the population 

(Item #3) In doing a survey of Chicago residents' evaluation of their schools, 
the researcher divided the city into 9-square-block areas and randomly 
selected 20 areas for his study. This is an example of: 

a. stratified sampling 

b. cluster sampling 

c. subgroup analysis 

d. use of a large sample 
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Two samples of students in introductory research methods classes were 
selected; one was used as the development sample (n=124) and the second as a 
research sample (n=85). A total of six classes were used in the study. One 
class being used as part of the development sample was tested at the 
beginning of the semester and two other classes were tested at midterm and at 
the end of the semester. The three classes forming the research sample were 
tested at midterm and at the end of th<? semester. The 93 test items were 
administered to the development sample and were subjected to item analysis; 
items were revised and administered to the research sample. The degree to 
which students mastered the competencies was assessed using the results 
from the research sample. Mastery was considered achieved for students who 
had correct answers on two of the three items for each of the 31 topics. 

Results 

The areas of mastery were primarily within the broad areas related to 
research design and process. Most topics not mastered by the students were 
within the general areas of descriptive statistics, data gathering, hypothesis 
testing, and inferential statistics. Table 1 illustrates the percent of students 
who mastered each of the 31 topics. Average mastery for each broad area was 
also calculated. On average, 87% of the students mastered research processes 
and 82% mastered research designs. However, in the other four broad areas, 
student competency was much less impressive. Students averaged 70% 
mastery on descriptive statistics, 61% succeeded in data gathering topics, 58% 
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mastered hypothesis testing items, and only 38% mastery was demonstrated 
in the inferential statistics area. Our findings indicated that most graduate 
students participating in this study, on average, did not master all of the 
necessary competencies to conduct independent, original research. 

Discussion and Conclusion 

The teaching faculty for research methods and statistics courses 
identified 31 topic areas as the essential knowledge base for master's level 
graduate students to acquire. The introductory research methods course that 
was required for master's students addressed each topic, although some topics 
received limited attention due to the extensive nature of the course content. 
Based on the results of this study, most students whose only training in 
research and statistics comes from this introductory course are possibly 
sufficiently trained to read and critically evaluate research in their fields, 
although these students would likely have difficulty interpreting a statistical 
analysis. Additionally, after completing an introductory research methods 
course, most students are ill-prepared to design, conduct, analyze, interpret, 
and write a report of a study. A traditional master's program requiring only 
one research course does not provide students with the opportunity to 
develop the needed skills so that they can participate, in a scholarly fashion, 
in activities related to research. 

In our ongoing research program, we will continue to assess graduate 
students' research skills. We will assess students' skills at varying stages of 
their scholarly development, in hopes that they will have the opportunity to 
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become involved in other research activities or take additional research and 
statistics courses. We will continue to work with teaching faculty in research 
and statistics courses to develop items to assess topic mastery; these items may 
be used for students to demonstrate their knowledge without taking the 
traditional research course. Also, from our topic list, we will work with 
faculty to develop a broad set of questions useful for written and oral 
comprehensive examinations. Our goal is to offer a set of standards to other 
professionals in the field for their reactions. Ultimately we hope to be able to 
communicate precisely and thoroughly the research and statistics 
competencies expected of graduate students by their faculty. 
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Topic Area Mastery 





Research Topic Areas 


Percent 

Mastering 


I. 


Research design 


82 % 


n. 


Research process 


87 % 




Formulation of viable research problems and hypotheses 


91% 




Scientific method 


94% 




Literature sources and searching 


89% 




Types of variables 


84% 




Types of sampling 


82% 




Ethical, legal, treatment of human subjects 


90% 




Internal and external validity of designs 


79% 


m. 


Hypothesis testing 


58 % 




Significance level, p-value, power 


46% 




Errors 


42% 




Null and alternative hypotheses 


79% 




Interpretation of results, statistical significance 


66% 


IV. 


Data gathering 


61 % 




Measurement 


59% 




Reliability estimates 


51% 




Validity estimates 


48% 




Standard error 


32% 




Conducting surveys and interviews 


81% 




Rating scales 


75% 




Likert scales 


79% 


V. 


Descriptive statistics 


70 % 




Dataset organization 


58% 




Measures of center 


89% 




Measures of dispersion 


70% 




Standard scores 


88% 




Contingency tables, chi-square 


61% 




Correlation coefficients 


48% 




Distributions 


74% 


VI. 


Inferential statistics 


38 % 




Elements of probability 


64% 




Sampling distributions 


42% 




Sampling error 


39% 




Common hypothesis testing procedures 


23% 




Interpreting computer printouts 


24% 



Note: Average percents for the six broad areas are in bold type. n=85. 
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